Endoscopic image enhancement system and method

ABSTRACT

The invention provides an image color enhancement system, method, storage medium and endoscope. Since the human skin color and the basic tone of endoscopic image are similar, a data set of Macbeth color card is utilized to generate a color correction matrix for skin color enhancement. The color correction matrix is used by an endoscope for real time image capture, resulting in a vivid expression of the basic tone of an endoscopic image without introducing artifacts in other tonal image parts.

TECHNICAL FIELD

The invention relates to the field of image processing, and inparticular the field of an endoscopic image color enhancement system,method and storage medium.

BACKGROUND

A human eye cannot directly see the digestive tract in its native state.The image captured by an endoscope has different color effects due tothe differences in devices and tuning methods by different manufacturersor end users. To date, there is no standard on image color rendering orenhancement in the endoscope industry. A generic image processor used inan endoscope provides adjustable hue and saturation options as a mean toenhance the image color to different flavors for different end users,which typically operates on the entire image by a set of same gainsresulting in an expected enhanced color in some parts of an image andunexpected appearances or artifacts in some other parts of the image. Asan example, a prominent endoscope provider Olympus of Japan has listedin its specifications of products the adjustable hue and saturationoptions.

SUMMARY OF THE INVENTION

To solve the above-mentioned limitation of the prior art, the presentinvention proposes an endoscopic image color enhancement system andmethod.

An endoscopic image color enhancement system, comprising:

An endoscope configured to: acquire a first image of the Macbeth colorcard under a standard light source matching a color temperature of theillumination of the endoscope, and adopt a color correction matrixgenerated based on the first image for skin tone enhancement to enhancecolor of endoscopic images; and

A data processing module configured to:

acquire a set of color enhancement target data, the set of colorenhancement target data including coordinates (H0, S0, V0) of a point P0in HSV space;

obtain data of the color patch of the first row and the first column ofthe Macbeth color card, including the coordinates (H1, S1, V1) of apoint P1 in HSV space;

obtain data of the color patches in the first row and second column ofthe Macbeth color card, including the coordinates (H2, S2, V2) of apoint P2 in the HSV space;

perform a first interpolation of the coordinates of P0 and thecoordinates of P1, then replace the coordinates of P1 with the result ofthe first interpolation, and perform a second interpolation of thecoordinates of P0 and the coordinates of P2, and then use the result ofthe second interpolation to replace the coordinates of P2;

merge data of the rest patches of Macbeth color card color with thereplaced P1 coordinates and the replaced P2 coordinates resulting in adata set of the Macbeth color card for skin tone enhancement;

generate the color correction matrix for skin color enhancement based onthe data set of the Macbeth color card for skin tone enhancement and thefirst image through a color correction matrix generation algorithm.

The data processing module is further configured to perform the firstinterpolation comprising:H10=a1*H0+a2*H1;S10=b1*S0+b2*S1;V10=c1*V0+c2*V1;

wherein, the value ranges of a1, a2, b1, b2, c1, and c2 are all [0,1];and

the second interpolation comprising:H20=a1*H0+a2*H2;S20=b1*S0+b2*S2;V20=c1*V0+c2*V2;

wherein, the value ranges of a1, a2, b1, b2, c1, and c2 are all [0, 1].

The data processing module is further configured to:

acquire a first target image of a best visual effect taken by theendoscope in a first subject;

calculate a basic tone of the first target image, including thecoordinates of a point in a HSV space resulting in the set of colorenhancement target data.

The data processing module is further configured to:

acquire a second target image taken by a preferred reference endoscopein the first or a second subject;

calculate a basic tone of the second target image, including coordinatesof a point in HSV space resulting in the set of color enhancement targetdata.

The data processing module is further configured to:

calculate a weighted average value of pixels of the first target imagewith wij being a weight of a pixel with coordinates of i and j, andconvert the weighted average value to HSV space resulting in the basictone of the first target image, wherein, wij is determined by:

converting the first target image to HSV space resulting in a firsttemporary image;

generating a first data set of a two-dimensional histogram of H and Sfrom the first temporary image;

perform a threshold filtering on the first data set to obtain a seconddata set of the two-dimensional histogram;

acquiring a centroid PCenter of the second data set;

calculating the distance dij from each pixel in the second data set toPCenter, wherein wij and dij are inversely proportional and wij equals 0for pixels not in the second data set and wij equals 0 for pixels not inthe second data set.

The data processing module is further configured to:

calculate a weighted average value of pixels of the second target imagewith wij being a weight of a pixel with coordinates of i and j, andconvert the weighted average value to HSV space resulting in the basictone of the second target image, wherein, wij is determined by:

converting the second target image to HSV space resulting in a firsttemporary image;

generating a first data set of a two-dimensional histogram of H and Sfrom the first temporary image;

perform a threshold filtering on the first data set to obtain a seconddata set of the two-dimensional histogram;

acquiring a centroid PCenter of the second data set;

calculating the distance dij from each pixel in the second data set toPCenter, wherein wij and dij are inversely proportional and wij equals 0for pixels not in the second data set and wij equals 0 for pixels not inthe second data set.

An endoscopic image color enhancement method implementable by a softwaremodule of a processor of an endoscope or a terminal device connected toan endoscope or having access to endoscopic images such as a display orviewing platform for diagnostic examination, comprises the followingsteps:

acquiring a set of color enhancement target data, the set of colorenhancement target data including coordinates (H0, S0, V0) of a point P0in HSV space;

obtaining data of color patches of first row and first column of theMacbeth color card, including the coordinates (H1, S1, V1) of a point P1in HSV space;

obtaining data of the color patches in the first row and second columnof the Macbeth color card, including the coordinates (H2, S2, V2) of apoint P2 in the HSV space;

performing a first interpolation of the coordinates of P0 and thecoordinates of P1, then replacing the coordinates of P1 with the resultof the first interpolation, and performing a second interpolation of thecoordinates of P0 and the coordinates of P2, and then using the resultof the second interpolation to replace the coordinates of P2;

merging data of the rest patches of Macbeth color card with the replacedP1 coordinates and the replaced P2 coordinates resulting in a data setof the Macbeth color card for skin tone enhancement;

generating a color correction matrix for skin color enhancement based onthe data set of the Macbeth color card for skin tone enhancement and thefirst image through a color correction matrix generation algorithm;

performing a color enhancement of an endoscopic image using the colorcorrection matrix.

A BRIEF DISCUSSION OF THE DRAWINGS

FIG. 1: Module diagram of the endoscopic image color enhancement system.

FIG. 2: Flow chart of the process for color enhancement of endoscopicimages.

FIG. 3: Flow chart of generating the color enhancement target data;

FIG. 4: An embodiment of an endoscope incorporating the colorenhancement functionality

DETAILED DESCRIPTION

An endoscope is preferably equipped with LED light source forillumination, preferably with a color temperature between 3000 and 7000k. The image color of the endoscope mainly depends on the illumination,the spectral characteristics of the objects in the scene, which is thedigestive tract of a patient as relating to the current invention, andthe image sensor and processor used to capture the image. The mucousmembrane is the main structure of the normal human digestive tract. Itis usually reddish and constitutes the base tissue for the basic tone ofthe endoscopic image. The basic tone or the predominant tone of anendoscopic image can be achieved preferably by calculating a weightedaverage of image pixels preferably in a sRGB color space, and preferablyconverting the weighted average from the sRGB space to HSV space. Othertones visible under the endoscope include bluish submucosal veins, foodresidues, digestive fluids, abnormal diseased tissues, and image noise.

A color correction matrix is a 2D data matrix comprising 3*3 elements. Acolor correction matrix operates on a pixel of an image typically in alinear R, G, B color space by the following formula:[R′,G′,B′]=A*[R,G,B]^(T).   [1]

Most imaging devices including the camera module in an endoscopyincorporate a CCM module in the ISP pipeline that operates in real timeon every pixel of an image acquired by the image sensor. Data of colorcorrection matrix are loaded to ISP hardware at the initialization anddynamically refreshable. Software based implementation of a colorprocessing of CCM operation is also applicable on various platforms. Toacquire a standard color correction matrix, a camera turns off its CCMmodule, takes a first image of a standard color card preferably theMacbeth color card under a standard illumination, wherein an colorcorrection matrix generation algorithm runs the image and a data set ofMacbeth color card to find out a set of optimal data which can performsformula [1] on the first image such that the output image data of thefirst image match a set of target data of the standard color card patchby patch within an acceptable range of error. There are many ways toimplement a color correction matrix generation algorithm in the priorart, and since the objective is not for inventing a novel colorcorrection matrix generation algorithm, there is no need to describe itin detail.

The color patches in the first row, first column and first row andsecond column of the Macbeth color card are the closest to and representthe human skin tone and are preferably referred to as the skin tonepatches hereby. It is a known technique in prior art to generate a colorcorrection matrix for enhancing the skin color by altering the values ofthe skin tone patches in a color correction matrix generation process.Due to a large amount of data showing that the color pick of the humandigestive tract under a preferred endoscope lighting color temperatureappears close to skin tone, it suggests enhancing the endoscopic imagejust like enhancing the skin color in a beauty camera.

As in FIG. 1, the endoscope 101, after turning off the working lightingof the endoscope and CCM function of its ISP, takes a first image of theMacbeth color card under a standard illumination of a color temperaturematching the working lighting color temperature of endoscope 101 oranother endoscope of the same model including having the same type ofimage sensor as 101. Preferably, the color temperature of the workinglighting of endoscope 101 may be 5000K; further, endoscope 101 adopts acolor correction matrix, which is generated by module 102 based on thefirst image, in the CCM module of the image processor for real-timeenhancement of the image captured by endoscope 101; A data processingmodule 102 is configured to: acquire a set of color enhancement targetdata, the set of color enhancement target data preferably includingcoordinates (H0, S0, V0) of a point P0 in the HSV space; Obtain standarddata of the color patches in the first row and the first column of theMacbeth color card, including preferably coordinates (H1, S1, V1) of apoint P1 in the HSV space; Obtain standard data of the color patches inthe first row and the second column of the Macbeth color card, includingpreferably coordinates (H2, S2, V2) of a point P2 in the HSV space;perform a first interpolation between the coordinates of P0 and thecoordinates of P1, then replaces the coordinates of P1 with the resultof the first interpolation; performs a second interpolation between thecoordinates of P0 and the coordinates of P2, then replaces thecoordinates of P2 with the result of the first interpolation; merge dataof the rest patches of standard data of the Macbeth color card colorwith the replaced P1 coordinates and the replaced P2 coordinates toobtain a data set of the Macbeth color card for skin color enhancement;generate the color correction matrix for skin color enhancement througha color correction matrix generation algorithm using the data set of theMacbeth color card for skin color enhancement and the first image whichis adopted preferably by endoscope 101 or another endoscope of the samemodel as endoscope 101 for endoscopic image enhancement in real time orby a software based color processing platform.

The first interpolation preferably comprises:H10=a1*H0+a2*H1;  [2]S10=b1*S0+b2*S1;  [3]V10=c1*V0+c2*V1  [4],wherein, the value of a1, a2, b1, b2, c1, and c2 are in [0, 1].The second interpolation preferably comprises:H20=a1*H0+a2*H2  [5];S20=b1*S0+b2*S2  [6];V20=c1*V0+c2*V2,  [7]wherein, the value of a1, a2, b1, b2, c1, and c2 are in [0, 1].

FIG. 3 is a flow chart for obtaining a set of color enhancement targetdata from an endoscopic image. Step 301 obtains an image by an endoscopein a first subject in a preview mode. Step 302 obtain multiple images inpreview mode by adjusting the hue and saturation parameters of thecamera of the endoscope. Step 303 captures an image with the best visualeffect as a first target image. Step 304 calculate a weighted averagevalue of pixels of the first target image with wij being a weight of apixel

with coordinates of i and j, wherein, wij is determined by:

converting the first target image to HSV space resulting in a temporaryimage;

generating a first data set of a two-dimensional histogram of H and Sfrom the temporary image;

performing a threshold filtering on the first data set resulting in asecond data set of the two-dimensional histogram;

acquiring a centroid PCenter of the second data set;

calculating the distance dij from each pixel in the second data set toPCenter, wherein

wij and dij are inversely proportional and wij equals 0 for pixels notin the second data set. Step 305 generates the set of color enhancementtarget data by converting the weighted average to the HSV color space.

FIG. 4 is a schematic diagram of an endoscope incorporating the systemor method of the endoscopic image color enhancement.

The embodiments described above are for the purpose of illustration, anyobvious changes derivative of this disclosure is claimed within theprotection scope of the present invention.

The invention claimed is:
 1. An endoscopic image color enhancementsystem, comprising: an endoscope configured to: acquire a first image,and use a color correction matrix generated based on the first image forskin tone enhancement to enhance color of endoscopic images; a dataprocessing module configured to: acquire a set of color enhancementtarget data, the set of color enhancement target data includingcoordinates (H0, S0, V0) of a point P0 in HSV space; obtain first dataof color patches in first row and first column of Macbeth color card,perform a first interpolation of the set of color enhancement targetdata and the first data, and replace the first data with result of thefirst interpolation; obtain second data of color patches in first rowand second column of Macbeth color card, and perform a secondinterpolation of the set of color enhancement target data and the seconddata, and replace the second data with result of the secondinterpolation; merge data of rest patches of Macbeth color card with thereplaced first and the replaced second data resulting in a data set ofMacbeth color card for skin tone enhancement; generate the CCM colorcorrection matrix for skin color enhancement based on the data set ofthe Macbeth color card for skin tone enhancement and the first imagethrough a color correction matrix generation algorithm.
 2. An endoscopicimage color enhancement method, comprising the steps of: acquiring afirst endoscopic image; acquiring a set of color enhancement targetdata, the set of color enhancement target data including coordinates(H0, S0, V0) of a point P0 in HSV space; obtaining first data of colorpatches in first row and first column of Macbeth color card, performinga first interpolation of the set of color enhancement target data andthe first data, and replacing the first data with result of the firstinterpolation; obtaining second data of color patches in first row andsecond column of Macbeth color card, performing a second interpolationof the set of color enhancement target data and the second data, andreplacing the second data with result of the second interpolation;merging data of rest patches of Macbeth color card with the replacedfirst and the replaced second resulting in a data set of Macbeth colorcard for skin tone enhancement; generating a CCM color correction matrixfor skin color enhancement based on the data set of the Macbeth colorcard for skin tone enhancement and the first endoscopic image through acolor correction matrix generation algorithm; enhancing color ofendoscopic images using the color correction matrix.